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Abstract 

Background: Necrotizing enterocolitis (NEC) is a major source of neonatal morbidity and mortality. Since there is no specific 
diagnostic test or risk of progression model available for NEC, the diagnosis and outcome prediction of NEC is made on 
clinical grounds. The objective in this study was to develop and validate new NEC scoring systems for automated staging 
and prognostic forecasting. 

Study design: A six-center consortium of university based pediatric teaching hospitals prospectively collected data on 
infants under suspicion of having NEC over a 7-year period. A database comprised of 520 infants was utilized to develop the 
NEC diagnostic and prognostic models by dividing the entire dataset into training and testing cohorts of demographically 
matched subjects. Developed on the training cohort and validated on the blind testing cohort, our multivariate analyses led 
to NEC scoring metrics integrating clinical data. 

Results: Machine learning using clinical and laboratory results at the time of clinical presentation led to two NEC models: (1) 
an automated diagnostic classification scheme; (2) a dynamic prognostic method for risk-stratifying patients into low, 
intermediate and high NEC scores to determine the risk for disease progression. We submit that dynamic risk stratification of 
infants with NEC will assist clinicians in determining the need for additional diagnostic testing and guide potential therapies 
in a dynamic manner. 

Algorithm availability: http://translationalmedicine.stanford.edu/cgi-bin/NEC/index.pl and smartphone application upon 
request. 
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Introduction 

Necrotizing enterocolitis (NEC) is one of the most common life- 
threatening diseases of the newborn and predominantly affects low 
birth weight infants in the first weeks of life with a reported 
frequency of between 1% and 5% of NICU admissions [1,2]. 
Published data from the NICHD neonatal network reports that 
7 % of very low birth-weight (VLBW < 1,000 gm.) infants develop 
definitive NEC (Bell's stage II) [3]. The pathogenesis of NEC 



includes progressive inflammation of the newborn gut involving 
enteric bacteria, the innate immune system and a compromised 
intestinal epithelial barrier resulting in eventual necrosis in 
advanced cases. In general, NEC can be thought of as occurring 
in two forms, medical (non-progressive) and surgical NEC 
(progressive). The extent of inflammation and intestinal injury in 
medical NEC is limited and reversible. In surgical NEC, the 
inflammatory process progresses resulting in irreversible necrosis 
and gangrene thus necessitating surgical resection of the affected 
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segments. Published overall mortality rates for NEC range from 
15% to 30%, and are higher in those infants requiring surgical 
intervention for progressive cases [4,5]. Several studies have 
demonstrated that the need for surgical intervention for NEC is an 
independent risk factor for long-term growth abnormalities, 
adverse neurodevelopmental outcomes and gastrointestinal mor- 
bidity including short bowel syndrome [5,6] . 

Bell's stage [7] is currently the most commonly utilized set of 
clinical criteria for diagnosing NEC. Bell's staging criteria are 
however susceptible to inter-observer subjectivity and overall do 
not forecast course of disease. In general, Bell's staging criteria are 
utilized to classify neonates under consideration of NEC as being 
suspected, confirmed and advanced. The determination of 
confirmed disease normally requires radiographic confirmation 
thus signifying intramural intestinal gas as a result of enteric 
bacterial invasion. The identification of neonates with progressive 
disease by clinical parameters alone generally occurs after a time 
when meaningful changes in patient management could signifi- 
cantly alter the course of disease. Taken together, these studies 
highlight the increasing biomedical burden imposed by NEC and 
therefore the need for better risk-stratification models of NEC in 
order to assist in altering its onset and progression. 

We postulated that NEC-associated clinical and laboratory data 
patterns could improve both the diagnostic accuracy and the 
clinical outcome prediction of infants under suspicion of having 
NEC. We utilized statistical learning methods to develop and 
validate NEC diagnostic and prognostic algorithms. We also 
propose the application of existing and evolving information 
technologies to facilitate adoption and deliver point-of-care 
application of our novel NEC predictive algorithm (Figure SI). 

Materials and Methods 

Subjects and associated clinical data sets 

This was a multi-institutional, multi-year study with prospective 
data collection performed from December 2003 to June 2011 by 
trained personnel at each participating institution. The time 
between study eligibility (i.e. clinical concern for NEC or at least 
Bell's stage I criteria) and declaration of confirmed NEC (non- 
progressive or medical) was recorded. Confirmed NEC in the 
Medical NEC cohort was defined as the presence of pneumatosis. 
Confirmation of Surgical NEC was defined at the time of surgery. 
The time from initial clinical suspicion to the progression to 
surgical disease averaged 2 days. This implies that a prognostic 
tool implemented at the time of initial clinical suspicion would 



potentially allow for the timely implementation of new or 
experimental management strategies. Informed consent was 
obtained from the parents of all enrolled subjects. This study 
was approved by the human subjects protection programs at each 
participating institute of the NEC consortium. The institutional 
review boards (IRBs) are the Institutional Review Board for 
Human Subject Research for Baylor College of Medicine and 
Affiliated Hospitals (BCM IRB), the Children's Hospital of 
Philadelphia Institutional Review Board, the Johns Hopkins 
Medicine Institutional Review Boards, Nationwide Children's 
Hospital Institutional Review Board, and the Human Research 
Protection Program. A binary decision tree (Figure 1) was 
constructed to post hoc classify the NEC consortium subjects into 
different stages according to Bell's staging criteria. Accordingly, all 
subjects in the NEC database were manually assigned to a specific 
Bell's stage according to the binary tree provided. Sixty-seven out 
of the 587 subjects were classified as Bell's stage Illb (pneumo- 
peritoneum). Since these subjects could represent another com- 
monly acquired neonatal intestinal disease (spontaneous intestinal 
perforations, SIP), they were excluded from further analysis. De- 
identified clinical and laboratory data were extracted for 
multivariate analysis. 

All subjects were randomized (R random sampling method) into 
cohorts for statistical training (2/3) and testing (1/3) analyses by 
categories of Bell's stages I, II, III (Table 1, training N = 343; 
testing N = 1 77). Demographic data was analyzed using R (http:// 
www.r-project.org/) epicalc package [8]. Chi-square test, Fisher's 
exact test and Kruskal-Wallis test were applied. All subjects' 
significant clinical datasets (27 parameters) were extracted and 
later used in the statistical learning to develop NEC staging and 
outcome scoring metrics. Significant historic factors include: 
"Feeding intolerance", "Apneic/bradycardic episode", "Oxygen 
desaturation episode", "Grossly bloody stools". Physical exam 
parameters included: "Abdominal distentions", "Capillary refill 
time (CRT) greater than 2 seconds", "Abdominal wall discolor- 
ation", "Abdominal tenderness". Medical history parameters 
included: "Was on a ventilator on the day he/she met the 
protocol definition of NEC", "Was on vasopressors on the day he/ 
she met protocol definition of NEC". Radiographic findings 
include: "Pneumatosis intestinalis", "Portal venous gas", "Ileus", 
"Dilated bowel", "Air/fluid levels", "Thickened bowel walls". 
Clinical laboratory data included: WBC ( x 1 0 / mm ); Neutrophils 
(%), ANC (absolute neutrophils count) (xl0 9 /L), bands (%), Band 
count (xl0 9 /L), Platelets (xl0 : VuL), Bicarbonate (meg/L), pH 



Ascites or Portal Venous Gas 




Ventilator or Vasopressors or Pneumatosis Intestinalis or 

Neutropenia at Day of Diagnosis Thicken bowel wall or Air/ 

i fluid levels 




Figure 1. A decision tree to guide the manual assignment of the modified Bell's staging criteria to the study subjects 

doi:1 0.1 371 /journal.pone.0089860.g001 
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Table 1. Demographics of NEC patients by Bell's staging criteria 







Training 








Testing 










Stage 1 


Stage II 


Stage III 


p-value 


Stage 1 


Stage II 


Stage III 


p-value 




N = 182 (53.1%) 


N = 1 32 (38.5%) 


N = 29(8.5%) 




N = 92 (52.0%) 


N = 66 (37.3%) 


N = 19 (10.7%) 




Male 1 


96 (52.7%) 


82 (62.1%) 


18 (62.1%) 


0.217 


52 (56.5%) 


33 (50.0%) 


14 (73.7%) 


0.184 


Race* 








0.023 








0.037 


White 


88 (48.4%) 


84 (63.6%) 


9 (31.0%) 




36 (39.1%) 


33 (50.0%) 


5 (26.3%) 




African American 


60 (33.0%) 


28 (21.2%) 


12 (41.4%) 




31 (33.7%) 


18 (27.3%) 


11 (57.9%) 




Asian 


6 (3.3%) 


3 (3.0%) 


0 (0%) 




5 (5.4%) 


0 (0%) 


0 (0%) 




Native Hawaiian or 


0 (0%) 


0 (0%) 


0 (0%) 




0 (0%) 


1 (1.5%) 


2 (10.5%) 




Pacific Islander 


American Indian or 


0 (0%) 


0 (0%) 


0 (0%) 




1 (1.1%) 


2 (3.0%) 


0 (0%) 




Alaskan Native 


Unknown 


23 (12.6%) 


12 (9.1%) 


6 (20.7%) 




17 (18.5%) 


10 (15.2%) 


1 (5.3%) 




Other 


5 (2.7%) 


4 (3.0%) 


2 (6.9%) 




2 (2.2%) 


2 (3.0%) 


0 (0%) 




Gestational Age (weeks} 5 


28 (26,31) 


31 (27,34) 


30 (27,31) 


<0.001 


28 (26,31) 


30.5 (27,33.8) 


28 (26.5,29) 


0.028 


Birth Weight (grams) 5 


941 (748.5,1437.5) 


1402 (937.2,2000.8) 


1110 (910,1540) 


<0.001 


931 (749.8,1428.8) 


1317.5 
(830,1746.2) 


1028 (875,1256.5) 0.063 


Birth Length [22] 5 


35 (31.2,40) 


38.5 (33,43) 


37 (32,39) 


0.004 


34.7 (32.5,40.1) 


38 (33,42) 


36 (33.8,38.8) 


0.249 


Birth Head Circumference 24.6 (22.6,27.5) 
[22] 5 


27.5 (23.2,30.5) 


26 (22.5,27.5) 


<0.001 


25.1 (23.2,28.5) 


26.8 (23.5,29.5) 


24.5 (23.1,26.8) 


0.173 


Medical/Surgical NEC* 








<0.001 








< 0.001 


Medical 


134 (73.6%) 


86 (65.2%) 


6 (20.7%) 




73 (79.3%) 


44 (66.7%) 


1 (5.3%) 




Surgical 


30 (16.5%) 


40 (30.3%) 


20 (69.0%) 




15 (16.3%) 


18 (27.3%) 


17 (89.5%) 




Missing 


18 (9.9%) 


6 (4.5%) 


3 (10.3%) 




4 (4.3%) 


4 (6.1%) 


1 (5.3%) 





1 Chi-square test is used. N is reported with percentages in parentheses. 
*Fisher's exact test is used. N is reported with percentages in parentheses. 
5 Kruskal-Wallis test is used. Median is reported with IQR in parentheses. 
doi:1 0.1 371 /joumal.pone.0089860.t001 



value, pH site and "Abdominal pain". Additional parameters of 
significance included acidosis when pH value < 7.3 and pH site is 
arterial or when bicarbonate value < 16, neutropenia when 
neutrophil count < 10 9 /L, and thrombocytopenia when platelet 
count < 100,000 per microliter. 

Development of the NEC staging scoring metric 

We performed the generalized linear mixed-effects models 
(GLMMs) [9] model analysis using R library function glmer from 
lme4 package [10] utilizing the 27 clinical parameters at each 
subject's initial clinical presentation. The NEC staging discrimi- 
nant function is the GLMM, and the method has the form 
response ~ terms where response is the ordinal factor (stage I, II, 
III) and terms are the 27 or subset of the 27 clinical parameters. 
Random effects of different subject hospital sites were considered 
during the training process of the GLMMs model. Regression 
coefficients for each of the variables along with the standard errors 
and i-values were computed. We also performed a Wald-test to 
evaluate the overall model fit using the R "lmtest" package [11]. 
Subsequently we reran the GLMMs model with the robust 
standard errors using R package "sandwich" [12] to adjust for 
heterogeneity in the model. This analysis yielded similar significant 
results, but gave more realistic /j-values. 9 of the original 27 
parameters were identified as significant and selected to construct 
the linear regression model. This 9-parameter subset included: 
"Abdominal pain", "Pneumatosis intestinalis", "Portal venous 
gas", "Dilated bowel", "Air/fluid levels", "Thickened bowel 



walls", "WBC (xloVmm 3 ) ", "Neutrophils (%)", and "Neutro- 
phil count". The computed value of the response variable as well 
as the "output" of the model was defined as the NEC staging 
score. Finally, we used the "predict" function to obtain the 
predicted values for the blind testing data set. 

Development of the NEC risk of disease progression 
metric 

36 patients were removed from the dataset (N = 520) due to 
incomplete data and inability to determine medical or surgical 
outcomes. Linear discriminant analysis (LDA) [13] was applied to 
stratify individual subjects based on clinical data. Specifically R 
library MASS [14] function "Ida" was utilized. Coefficients of 
linear discriminants (LD1) were calculated as a measure of the 
association of each variable with the final diagnosis. LDA created 
linear combinations of these clinical variables and calculated 
coefficients LD1 to optimize separation between medical NEC 
and surgical NEC groups. The NEC outcome discriminant 
function is the LDA, and the predictive method has the form 
response ~ terms where response is the factor (progressive and 
non-progressive NEC) and terms are the 27 or subset of the 27 
clinical parameters. To gauge the performance of the NEC 
outcome LDA model to classify non-progressive from progressive 
NEC subjects, we performed receiver operating characteristic 
(ROC) analyses [15]. 

The LDA discriminant scoring metric, designated as the NEC 
outcome score, enables the clinical variables to be collectively 
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interpreted on a scale, rather than a strict binary discrimination. 
We calculated NEC outcome scores based on clinical findings (27 
parameters) for all subjects in the training cohort and stratified 
them into subgroups with low, intermediate, and high scores based 
on 90% correct classification in the low (10% probability of 
medical NEC) and high risk (90% probability of surgical NEC) 
groups. 

Results 

Demographics 

In both training and testing data sets (Table 1), race, gestational 
age, birth weight, length and head circumference related 
differences were observed between different subjects according 
to their Bell's stage. Statistical differences among Bell's stage I, II 
and III NEC were observed in race (p-value: training 0.023 and 
testing 0.037), gestation age (/>-value: training < 0.001 and testing 
0.028), birth weight (p-value: training < 0.001 and testing 0.063), 
birth length (/(-value: training 0.004 and testing 0.249), and birth 
head circumference (jft-value: training < 0.001 and testing 0.173). 
No statistical differences were observed in gender among the 
different NEC stages. 

For the risk of disease progression model, there were differences 
in gender, gestational age, birth weight, length and head 
circumference between different subjects according to their 
progressive/ non-progressive designation and outcome. In progres- 
sive NEC, there were slightly more male infants (/>-value: training 
0.068 and testing 0.024), smaller gestational age (/>-value: training 
0.067 and testing 0.037), smaller birth weight (p-value: training 
0.09 and testing 0.095), smaller birth length (p-value: training 
0.132 and testing 0.033), and smaller birth head circumference (p- 
value: training 0.002 and testing 0.005). No statistical differences 
were observed for race between progressive and non-progressive 
NEC. 

Automation of the NEC Bell's Staging Assignment 

Recognizing that Bell's stage requires significant clinician input 
to accurately classify patients as was done in Figure 1 , we initially 
sought to create a model that could diagnose NEC according to 
presenting stage without the need for manual assignment. This 
effort was further motivated by a desire to remove the subjective 
nature of current NEC diagnostic algorithms and as a prelude to 
automating these tasks to accommodate real-time computation. 
We hypothesized that an automated staging system similar to the 
Bell's staging criteria could be developed using clinical data at the 
time of presentation. Utilizing a 9-parameter model (Figure 2), 
subjects were objectively assigned a stage at the time of 
presentation thus avoiding the need for manual assignment. 
The, horizontal lines in Figure 2 designate the scores where 90% 
of stage I (or stage III) subjects were classified correctly and 
matched the original manual Bell's stage designation (Figure 1) of 
the training subjects. The computed NEC stage demonstrated 
promising agreement with the clinical categorical assignment: 
Stage I (training: 100%, testing: 100%), Stage II (training: 97.7%, 
testing: 94.0%), Stage III (training: 89.7%, testing: 83.3%). In both 
training and testing, none of the Stage I or III patients was 
mistakenly cross-assigned to Stage III or I respectively. 

Development of NEC outcome scoring metrics 

In order to determine whether statistical modeling could be 
used to help predict patient outcome, a strategy of risk 
stratification rather than binary prediction was pursued. An 
LDA based model (Figure 3) was used to risk stratify the training 
and testing subjects into 3 levels of risk for progression of NEC 



(Figure 4 and Figure S2): low (score < 0.1313; in training 
cohort: 187 medical NEC and 20 surgical NEC; in testing cohort: 
90 medical NEC and 10 surgical NEC testing subjects), 
intermediate 0.1313^ score S 2.2160; in training cohort: 40 
medical NEC and 54 surgical NEC; in testing cohort: 25 medical 
NEC and 26 surgical NEC) and high (score > 2.2160; in training 
cohort: 2 medical NEC and 20 surgical NEC; in testing cohort: 0 
medical NEC and 10 surgical NEC). The intermediate groups 
include 29.1% and 31.7% of the total training and testing subjects 
respectively whose outcome could not be confidently predicted by 
our clinical scoring algorithm. ROC AUCs of the clinical score for 
a determination of risk of NEC progression were 0.84 and 0.85 in 
training and testing cohorts respectively. With this NEC outcome 
scoring metric, we can effectively stratify the NEC subjects 
according to their NEC progressive risk using criteria obtainable at 
the time of disease presentation, resulting in 80.5% of the medical 
subjects (N = 277) assigned with 10% risk of being progressive, and 
21.4% of the surgical NECs (N = 30) assigned with 90% risk of 
being progressive. Overall, although our predictive method 
yielded encouraging positive predictive rates (PPV) of 90.2% and 
93.8% for accurate (90% confidence interval) medical and surgical 
NEC outcome prediction respectively, there were still 18.9% 
medical NEC and 57.1% surgical NEC subjects that remained 
undetermined, and 0.6% of medical NEC and 21.4% surgical 
NEC subjects in-correctly predicted. 

Next, we set to explore the impact on NEC outcome prediction 
incurred by a reduction in the number of clinical parameters 
available for risk stratifying. We reasoned that many potential 
patients would have incomplete data in the early presentation or 
suspicion of disease. The practical consideration is that when 
information of a particular clinical parameter is unavailable, the 
algorithm can still provide a useful determination of risk. 
Beginning with the smallest-weighted parameter (Figure 3, right 
to the left), we iteratively reduced the number of parameters to 
construct outcome prediction models of increasingly smaller panel 
sizes. The models were developed using training data and tested 
blindly on the testing dataset. AUCs were computed and ROC 
curves plotted (Figure 5). AUC analyses indicated that the model 
performance deteriorates when the model's panel size is less than 7 
parameters. 

Discussion 

The clinical presentation of neonates with signs and symptoms 
suggestive of NEC can be obscured by similarities to other 
inflammatory conditions of the newborn. Moreover, there are no 
reliable prognostic signs of progressive NEC that herald disease 
progression prior to patients reaching an irreversible state 
requiring surgery. Clinicians have therefore utilized combinations 
of clinical indicators and experience to guide patient management. 
Specifically, NEC is diagnosed through a combination of clinical, 
radiographic and laboratory findings that in aggregate define the 
original Bell's staging criteria. We automated a NEC stage 
assignment at diagnosis according to a 9-feature panel with utility 
comparable to the manual assignment of Bell's staging criteria. 
Overall there was generally good agreement between the 
computed and manually assigned stages for the lower stage groups 
(100% testing) upon presentation. The correlation between the 
computed and manual stage deteriorated with increasing stage, 
however (Stage II 94.0% and Stage III 83.3% testing). Although 
Bell's staging assignment at the time of initial diagnosis is useful, it 
is a static diagnostic algorithm that was not intended to forecast 
disease progression. Our data (Figure S3) reveal that the majority 
of patients that develop progressive NEC do not present with a 
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• ill 


1 


• II 


• ill 




1 


180 (177, 184) 


0 (0, 0) 


0 (0, 0) 


94 (90, 97) 


0 (0, 0) 


0 (0, 0) 


Computed 


II 


0(0,0) 


126 (123, 130) 


4 (4, 5) 


0 (0, 0) 


65 (61, 69) 


2 (1, 3) 




III 


0 (0, 0) 


4 (4, 5) 


28 (26, 30) 


0 (0, 0) 


3 (2, 3) 


14 (12, 16) 



Figure 2. Automated NEC staging assignment results. Left: modeling training. Right: blind testing. Bottom: manual versus automated NEC 
staging assignment comparative analysis. To gauge the impact of different training/testing cohort partition on the statistical learning, we performed 
a bootstrapping analysis that randomly partitioned the cohorts into 100 different training/testing sets. Results were summarized where median and 
interquartile range (IQR) values were calculated for each comparative category. 
doi:1 0.1 371 /journal.pone.0089860.g002 




Figure 3. Clinical variable's contribution (LD1) to the NEC outcome LDA model. LDA: Linear discriminant analysis. LD1: first discriminant 
variable. 

doi:1 0.1 371 /journal.pone.0089860.g003 
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Figure 4. NEC outcome predictive results. A. ROC AUC analysis. To gauge the impact of different training/testing cohort partition on the 
statistical learning, we performed a bootstrapping analysis that randomly partitioned the cohorts into 100 different training/testing sets. The 
distribution of 100 ROC curves, training and testing respectively, are illustrated. B. Use of the NEC outcome prediction metric to risk-stratify NEC 
subjects into low, intermediate and high risk groups. 
doi:1 0.1 371 /journal.pone.0089860.g004 



fulminate course and would therefore benefit from a better 
prediction model. 

Bell's original staging criteria and subsequent modifications 
have been used by clinicians to apply diagnostic labels to patients 
as being either "suspicious", "confirmed" or representing 
"advanced stage" NEC. Additional follow-up measurements, 
including frequent abdominal radiographs and laboratory values 
are obtained in an attempt to define further patient trajectory of 
disease. A label of confirmed NEC normally requires a finding of 
pneumatosis intestinale on an abdominal radiograph in an infant 
that has other systemic signs and symptoms suggestive of illness. 
Although pneumatosis is diagnostically definitive, it does not 
predict outcome or suggest a specific therapy. The label of 
"advanced NEC" implies that the disease has progressed to a 
point of irreversible intestinal injury including intestinal necrosis 
producing perforation and thus requiring an operation to excise 
the affected intestine. Unfortunately, the designation of progressive 
NEC by utilizing hard clinical signs only occurs at a time when the 
disease has already advanced and there remains little opportunity 
to offer novel treatment options that might alter the course of 
disease. 



The definition of NEC was important for this study and was 
implemented in such a way that our datasets did not contain 
infants with potential SIP. SIP is a separate and distinct disease 
entity. In many national datasets, the surgical NEC subsets are as 
much as 20% SIP, and in our original database there was 
ambiguity in the diagnosis of 67/587 infants given the initial 
presentation of pneumoperitoneum. Not modeling to predict NEC 
associated death is a limitation of the study, which clearly should 
be addressed in the future when more comprehensive datasets are 
available. There were 75 deaths amongst NEC subjects across the 
five children's hospitals in our database between 2003 and 2010. 
Unfortunately among these subjects, most of the needed clinical 
parameters were not collected. 

Utilizing this curated large multicenter prospective clinical 
database, we sought to develop a prediction model for the risk of 
NEC progression. We reasoned that NEC risk stratification would 
find the most clinical utility in patients presenting without an 
obvious clinical trajectory. Initially, in order to build a risk of 
disease progression model we utilized 27 available data points 
recorded at the time of initial clinical suspicion of NEC. These 
data points were used to construct a tiered risk stratification 
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Figure 5. NEC outcome predictive LDA models with reduced 
number of variables (listed in descending order from right to 
left in Figure 3 by the absolute value of their weights). The 

model performance was gauged by ROC analysis. Vertical dotted line: 
the model performance deteriorates when the model's panel size is less 
than 7 parameters. 
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scheme in order to increase its flexibility and improve upon the 
binary (or categorical) classification suggested by Bell's staging 
criteria (i.e. advanced disease or something less). Moreover, a 
majority of subjects with low risk disease (80.5%) can be positively 
identified with high confidence (90% CI) while only a small 
percentage (21.4%) of subjects that were found to have eventual 
progressive disease could be identified a priori and at the time of 
initial presentation with a high degree of confidence (90% CI). 
Thus, despite a good positive predictive value for the very low and 
high-risk groups identified by this algorithm, there remains 18.9% 
of the medical and 57.1% of the surgical subjects that are 
indeterminate and would require further testing. 

Not surprisingly, the previously described NEC parameters of 
metabolic acidosis (pH) [16-19] and portal venous gas (PVG) [16- 
18] were found by our multivariate machine learning to be the 
most weighted predictors (Figure 3) on the NEC outcome score. 
This congruency suggests that our machine learning methodology 
identified both statistically and pathophysiologically significant 
predictors of NEC. We speculate that this type of risk stratification 
will provide several relevant clinical applications to be determined 
in future studies for either identifying patients for additional 
diagnostic/prognostic testing or perhaps alternative therapies (i.e. 
earlier surgery). Despite the simplicity of design, this type of risk 
stratification with an AUC of 0.85 testing by ROC analyses was 
superior to Bell's staging criteria in projecting patient outcomes 
with NEC. Still, there remained a full 30.0% of patients found to 
be of intermediate risk that would require additional testing or a 
novel test to help further define these patients' risk of progression. 
This is one of the principle limitations of our study that we posit 
can only be resolved through the discovery and addition of 
additional features including possible molecular biomarkers to the 
model. Due to this limitation, caution should be applied in using 
our proposed risk stratification model to guide management of 
infants with NEC. Our work suggests future NEC outcome 



biomarker studies should focus on the discovery of markers that 
can differentiate NEC subjects with intermediate risk scores for 
progression. We speculate that biomarker testing that is in 
development by several groups including our own may address 
this unmet need [20,21]. It also remains to be determined whether 
iterative longitudinal testing of patients with an obscure clinical 
course might be further instructive using the current clinical 
parameter model. 

In order to facilitate further clinical validation, we have also 
sought to determine the performance of the progression risk model 
using incomplete data. We reasoned that in the clinical setting 
many patients would be unlikely to have complete datasets to fulfill 
model requirements and achieve risk stratification. Our iterative 
testing and ROC curve analysis demonstrated that the model 
maintains robust performance characteristics over a wide range of 
values. This is encouraging and highlights an additional advantage 
of this algorithm given its ability to accommodate missing values 
while still providing a useful output. By comparing a new patient 
to all of the patient data points in the database one can determine 
where the new patient clusters in an unsupervised analysis. 

A challenge to deriving predictive algorithms is to show utility 
relative to the existing status quo. As any predictive model will be 
most useful if able to reveal an obscure disease or forecast an 
unforeseen event, our models hold significant potential for clinical 
utility relative to currently employed methods of NEC diagnosis 
and prognosis. With ongoing refinement, these algorithms may 
accurately identify low-risk patients and perhaps enable imple- 
mentation of novel therapeutic approaches for high-risk patients as 
they are developed. 

Supporting Information 

Figure SI Point-of-care applications of proposed NEC algo- 
rithm. To allow point of care utilization of our innovative NEC 
algorithm, we designed and created a NEC analytical website 
application at http:// translationalmedicine.stanford.edu/ cgi-bin/ 
NEC/nec.pl to permit remote diagnostic and prognostic guidance, 
based on patient demographics, historic factors, physical exam, 
medical history, fluid intake and nutrition, radiographic findings, 
laboratory tests, and other findings. The simplified web applica- 
tion was built and hosted at: http://translationalmedicine. 
stanford.edu/cgi-bin/NEC/index.pl, which is based on the 
aforementioned feature selection result. In addition, a smartphone 
application was developed to support point of care NEC 
management through the utilization of NEC clinical database 
and algorithms. Implementation technical details: with a Model 
View Controller (MVC) design, we applied a previously developed 
server-based bio-computational framework [23] to allow physi- 
cians remote access to our NEC algorithms. The algorithm and 
associated web application were implemented using R and PERL 
(http://www.perl.org/) respectively. The iPhone application was 
developed in the integrated development environment [24] Xcode 
4.6.2 and iOS SDK 6.1.2 using Objective C. Later it was tested 
using iPhone Simulator 6.1, iPhone 5 and new iPad, it was 
supported by iOS 6. 1 or higher versions. Recognizing the need for 
automation to facilitate adoption and to potentially shield 
clinicians from the complex details and computation required to 
risk stratify patients using these algorithms, we created server- 
based applications that are capable of providing immediate risk 
stratification via a graphical output to users. We speculate that this 
type of mobile application and server-based computation will 
facilitate robust validation and additional longitudinal testing of 
the algorithms provided. We also submit that this type of 
electronic application will promote adoption of the existing 
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algorithms while promoting the evolution and improved perfor- 
mance of the existing prediction models through the addition of 
additional subjects data. 
(TIF) 

Figure S2 Results of the risk stratification analysis for the 

progression of NEC. 

(TIF) 

Figure S3 Exploratory correlation analysis between NEC 
outcome and the assigned Bell's staging determined at initial 
clinical presentation. A. Density plots of Bell's stage I, II, III 
patients relative to NEC outcome (medical, surgical NEC). The 
density plot of Bell's stage I-III subjects was analyzed in terms of 
NEC outcomes (medical, surgical). The medical NEC patient 
number negatively and the surgical NEC patient number 
positively correlated with the increment of Bell's stage: 32.1%, 
41.4% and 26.4% of the progressive (surgical) NEC subjects were 
in NEC staging I, II and III while 52.1%, 38.8% and 9.1% of aU 
NEC subjects were in NEC staging I, II and III respectively. 
Among all stage III patients, 84.1% were ultimately progressive 
NEC and had subsequent surgery. Assuming all stage III patients 
are progressive, the positive predictive value is 84. 1 % and negative 
predictive value is 76.6%. However, this algorithm failed to 
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